[axonius][identity] Add Axonius Identity datastream by muskan-agarwal26 · Pull Request #16620 · elastic/integrations

muskan-agarwal26 · 2025-12-18T12:20:41Z

Proposed commit message

The release includes identity data stream and associated dashboard.

Axonius fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from live data samples, which were subsequently
sanitized.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

To test the axonius package:

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/axonius directory.
Run the following command to run tests.

elastic-package test

2025/12/18 17:48:46  INFO New version is available - v0.117.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.117.1
Run asset tests for the package
2025/12/18 17:48:46  INFO License text found in "/root/GITHUB/integrations/LICENSE.txt" will be included in package
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │             │ asset     │ dashboard axonius-60472232-ca7b-45e6-9fa6-72e6efc41a8e is loaded │ PASS   │      1.422µs │
│ axonius │ identity    │ asset     │ index_template logs-axonius.identity is loaded                   │ PASS   │        348ns │
│ axonius │ identity    │ asset     │ ingest_pipeline logs-axonius.identity-0.1.0 is loaded            │ PASS   │        233ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run pipeline tests for the package
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬─────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                           │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼─────────────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │ identity    │ pipeline  │ (ingest pipeline warnings test-account.log)         │ PASS   │ 778.105629ms │
│ axonius │ identity    │ pipeline  │ (ingest pipeline warnings test-certificate.log)     │ PASS   │ 700.415371ms │
│ axonius │ identity    │ pipeline  │ (ingest pipeline warnings test-group.log)           │ PASS   │ 662.882359ms │
│ axonius │ identity    │ pipeline  │ (ingest pipeline warnings test-identity-common.log) │ PASS   │ 763.614554ms │
│ axonius │ identity    │ pipeline  │ (ingest pipeline warnings test-user.log)            │ PASS   │ 639.719729ms │
│ axonius │ identity    │ pipeline  │ test-account.log                                    │ PASS   │ 296.335698ms │
│ axonius │ identity    │ pipeline  │ test-certificate.log                                │ PASS   │ 309.327719ms │
│ axonius │ identity    │ pipeline  │ test-group.log                                      │ PASS   │ 308.939347ms │
│ axonius │ identity    │ pipeline  │ test-identity-common.log                            │ PASS   │ 291.192435ms │
│ axonius │ identity    │ pipeline  │ test-user.log                                       │ PASS   │ 316.869779ms │
╰─────────┴─────────────┴───────────┴─────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run policy tests for the package
--- Test results for package: axonius - START ---
No test results
--- Test results for package: axonius - END   ---
Done
Run static tests for the package
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ axonius │ identity    │ static    │ Verify sample_event.json │ PASS   │ 268.878426ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run system tests for the package
2025/12/18 17:48:59  INFO Installing package...
2025/12/18 17:48:59  INFO License text found in "/root/GITHUB/integrations/LICENSE.txt" will be included in package
2025/12/18 17:49:01  INFO Running test for data_stream "identity" with configuration 'default'
2025/12/18 17:49:09  INFO Setting up independent Elastic Agent...
2025/12/18 17:49:20  INFO Setting up service...
2025/12/18 17:49:40  INFO Validating test case...
2025/12/18 17:49:42  INFO Tearing down service...
2025/12/18 17:49:43  INFO Write container logs to file: /root/GITHUB/integrations/build/container-logs/axonius-1766060383822768504.log
2025/12/18 17:49:47  INFO Tearing down agent...
2025/12/18 17:49:48  INFO Write container logs to file: /root/GITHUB/integrations/build/container-logs/elastic-agent-1766060388442966883.log
2025/12/18 17:50:15  INFO Uninstalling package...
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ axonius │ identity    │ system    │ default   │ PASS   │ 41.623316269s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: axonius - END   ---
Done

Related issues

Screenshots

elastic-vault-github-plugin-prod · 2025-12-18T16:24:59Z

Package microsoft_exchange_online_message_trace - 1.29.1 containing this change is available at https://epr.elastic.co/package/microsoft_exchange_online_message_trace/1.29.1/

botelastic · 2026-02-07T03:40:34Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2026-03-09T03:56:53Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

botelastic · 2026-04-08T15:17:23Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

…ntegrations into datastream-identity

efd6 · 2026-04-20T01:40:17Z

+                    :
+                      string(resp.Status) + " (" + string(resp.StatusCode) + ")"
+                  ),
+                  "asset_type": string(state.worklist.asset_type_list[0]),


Why does this need a string conversion?

No need, removing string conversion for this from everywhere in code.

efd6 · 2026-04-20T01:58:45Z

+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


Check that all date and convert processors will remove the field value on failure. This field is defined as a date so if it fails we have left an unmappable value in place.

efd6 · 2026-04-20T02:17:12Z

Can this be more concisely/manageably expressed with some/most of the processors replaced with a script

efd6 · 2026-04-20T02:24:32Z

+- version: 0.1.1
+  changes:
+    - description: Add support for identity data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16620


This should be in the 0.1.1 changes. This is not being merged into main in this PR.

efd6 · 2026-04-20T02:24:40Z

 name: axonius
 title: Axonius
-version: 0.1.0
+version: 0.1.1


Suggested change

version: 0.1.1

version: 0.1.0

efd6 · 2026-04-20T02:30:00Z

+    "id": "axonius-60472232-ca7b-45e6-9fa6-72e6efc41a8e",
+    "references": [
+        {
+            "id": "metrics-*",


efd6 · 2026-04-20T02:30:10Z

+            "type": "index-pattern"
+        },
+        {
+            "id": "metrics-*",


logs? Also, duplicate of the index-pattern at L1275.

efd6 · 2026-04-20T02:36:35Z

+        {
+            "id": "logs-*",
+            "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index",
+            "type": "index-pattern"
+        }


Duplicate of pattern at L1279.

macroscopeapp · 2026-04-21T12:50:51Z

+      name: '{{ IngestPipeline "pipeline-account" }}'
+      tag: pipeline-account
+      if: >-
+        ctx.axonius?.identity?.asset_type.contains('accounts')


🟠 High ingest_pipeline/default.yml:605

The pipeline routing conditions on lines 605, 610, 615, and 620 use ctx.axonius?.identity?.asset_type.contains(...) which throws a NullPointerException when asset_type is null. The optional chaining only guards axonius and identity; once identity exists, the expression evaluates to null.contains(...) when asset_type is missing. This causes the entire pipeline to fail and fall through to on_failure, leaving documents incompletely processed. Consider adding null guards like ctx.axonius?.identity?.asset_type != null && ctx.axonius.identity.asset_type.contains('accounts') for all four conditions.

- ctx.axonius?.identity?.asset_type.contains('accounts') + ctx.axonius?.identity?.asset_type != null && ctx.axonius.identity.asset_type.contains('accounts')

🤖 Copy this AI Prompt to have your agent fix this:

In file packages/axonius/data_stream/identity/elasticsearch/ingest_pipeline/default.yml around line 605: The pipeline routing conditions on lines 605, 610, 615, and 620 use `ctx.axonius?.identity?.asset_type.contains(...)` which throws a `NullPointerException` when `asset_type` is null. The optional chaining only guards `axonius` and `identity`; once `identity` exists, the expression evaluates to `null.contains(...)` when `asset_type` is missing. This causes the entire pipeline to fail and fall through to `on_failure`, leaving documents incompletely processed. Consider adding null guards like `ctx.axonius?.identity?.asset_type != null && ctx.axonius.identity.asset_type.contains('accounts')` for all four conditions.

This assessment is correct; there is no previous condition or action that guarantees that asset_type is not absent or null. An alternative fix is to use ctx.axonius?.identity?.asset_type?.contains('accounts') == true.

Thank you for confirming the issue and suggesting the alternative approach. The ?.contains(...) == true pattern is indeed cleaner and more concise.

Would you like me to fix all four conditions (lines 605, 610, 615, and 620) using this approach?

Leave all fixes for the author.

Understood. I'll leave this for the author to fix.

Keeping it as is.

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

@muskan-agarwal26 This should be fixed.

macroscopeapp · 2026-04-21T12:50:51Z

+              type: boolean
+              description: Indicates if auto-delete days for cloud recordings is configured.


🟡 Medium fields/fields.yml:971

axonius.identity.recording.auto_delete_cmr_days is defined as type: boolean, but the field name indicates it should store a numeric day count. When the Zoom API sends an integer like 30, the ingest pipeline's boolean conversion coerces it to true, silently dropping the actual value. Consider changing the type to long and removing this field from the boolean conversion script in pipeline-user.yml.

- name: auto_delete_cmr_days - type: boolean + type: long

🤖 Copy this AI Prompt to have your agent fix this:

In file packages/axonius/elasticsearch/transform/latest_identity/fields/fields.yml around lines 971-972: `axonius.identity.recording.auto_delete_cmr_days` is defined as `type: boolean`, but the field name indicates it should store a numeric day count. When the Zoom API sends an integer like `30`, the ingest pipeline's boolean conversion coerces it to `true`, silently dropping the actual value. Consider changing the type to `long` and removing this field from the boolean conversion script in `pipeline-user.yml`.

Do we have a test that exercises this? Is it documented somewhere? I don't see it in any tests.

No tests currently exercise this field. I found it in the boolean conversion script at pipeline-user.yml line 775, where it's listed alongside other recording.* fields to be converted to boolean. The field is documented as type boolean with description "Indicates if auto-delete days for cloud recordings is configured." However, the field name auto_delete_cmr_days suggests it should store the actual day count (e.g., 30, 60, 90), not just a boolean flag—that's what the companion field auto_delete_cmr is for.

The comment was actually directed to @muskan-agarwal26.

yes, in live logs, we are getting value as true or false, hence it is kept boolean

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

@muskan-agarwal26 Please make it so that this is exercised in our pipeline tests.

efd6

Please address https://github.com/elastic/integrations/pull/16620/changes#r3107948170 and the macroscopeapp concerns.

muskan-agarwal26 · 2026-04-22T14:16:26Z

@efd6 I have addressed the comment: https://github.com/elastic/integrations/pull/16620/changes#r3107948170 , used script processors to reduce the count of processors.

efd6 · 2026-04-23T02:21:03Z

I have addressed the comment: https://github.com/elastic/integrations/pull/16620/changes#r3107948170 , used script processors to reduce the count of processors.

I don't see any change in the size of that file.

muskan-agarwal26 · 2026-04-23T08:53:21Z

I have addressed the comment: https://github.com/elastic/integrations/pull/16620/changes#r3107948170 , used script processors to reduce the count of processors.

I don't see any change in the size of that file.

It is actually reduced, it can be viewed in this commit: 8b75b00

muskan-agarwal26 · 2026-04-24T05:27:41Z

@efd6 , please have a look at my comment: #16620 (comment)

muskan-crest added 2 commits December 18, 2025 17:48

Added identity datastream in Axonius

27dcab4

Update changelog entry

0222d3a

muskan-agarwal26 changed the base branch from main to feature/axonius-0.1.0 December 18, 2025 12:22

Added transform pipeline

16cee79

botelastic Bot added the Stalled label Feb 7, 2026

botelastic Bot closed this Mar 9, 2026

cpascale43 reopened this Mar 9, 2026

botelastic Bot removed the Stalled label Mar 9, 2026

botelastic Bot added the Stalled label Apr 8, 2026

cpascale43 removed the Stalled label Apr 8, 2026

Merge branch 'feature/axonius-0.1.0' of github.com:muskan-agarwal26/i…

ab89232

…ntegrations into datastream-identity

muskan-agarwal26 marked this pull request as ready for review April 19, 2026 14:11

muskan-agarwal26 requested a review from a team as a code owner April 19, 2026 14:11

macroscopeapp Bot reviewed Apr 19, 2026

View reviewed changes

efd6 reviewed Apr 20, 2026

View reviewed changes

andrewkroh removed the New Integration Issue or pull request for creating a new integration package. label Apr 20, 2026

Resolved comments provided by bot and @efd6.

8b75b00

macroscopeapp Bot reviewed Apr 21, 2026

View reviewed changes

muskan-agarwal26 requested a review from efd6 April 21, 2026 12:57

efd6 reviewed Apr 22, 2026

View reviewed changes

Comment thread packages/axonius/kibana/dashboard/axonius-60472232-ca7b-45e6-9fa6-72e6efc41a8e.json

Resolved further comments of @efd6 and bot.

2bfe617

muskan-agarwal26 requested a review from efd6 April 22, 2026 14:17

andrewkroh removed dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:axonius [Integration not found in source] labels Apr 23, 2026

github-actions Bot mentioned this pull request Apr 24, 2026

[oblt-aw][agent-suggestions] Agent suggestions - 2026-04-24 elastic/oblt-actions#916

Open

4 tasks

		type: boolean
		description: Indicates if auto-delete days for cloud recordings is configured.

Conversation

muskan-agarwal26 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod Bot commented Dec 18, 2025

Uh oh!

botelastic Bot commented Feb 7, 2026

Uh oh!

botelastic Bot commented Mar 9, 2026

Uh oh!

botelastic Bot commented Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

macroscopeapp Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

macroscopeapp Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

macroscopeapp Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muskan-agarwal26 commented Dec 18, 2025 •

edited

Loading